The Essence of Pose

Convolutional Neural Networks (CNNs) are often used to extract a lower-dimensional feature descriptor from images. Though they have recently been deployed successfully to solve tasks in computer vision such as semantic segmentation, image reconstruction or depth understanding, the inner workings of CNNs are still subject to current research. Recent research to better understand features learned by CNNs has focused on networks trained to do image classification. We perform a similar investigation, but in the realm of 3D pose estimation. We explore inversion techniques similar to those in [12, 13, 3] to invert the feature descriptor learned by PoseNet, a CNN by the Stanford Computer Vision and Geometry laboratory trained to understand the 3D pose of objects. We use a dataset consisting of several million images annotated with camera parameters. Our results suggest that PoseNet mainly embeds information about edges that point towards the main vanishing point in the feature descriptor. Most strikingly, color and texture information are lost, while the feature descriptor retains enough information to reconstruct the rough shape of the object in the original image.